Multi-parameter adaptive loop filtering in video processing

ABSTRACT

Devices, systems and methods for video processing are described. In an exemplary aspect, a method for video processing includes performing a conversion between a coded representation of a video comprising one or more video regions and the video. The coded representation includes first side information that provides a clipping parameter for filtering a reconstruction of a video unit of a video region using a non-linear adaptive loop filter, and wherein the first side information is signaled together with second side information indicative of filter coefficients used in the non-linear adaptive loop filter.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/CN2020/080829, filed on Mar. 24, 2020 which claims the priorityto and benefits of International Patent Application No.PCT/CN2019/079395, filed on Mar. 24, 2019. All the aforementioned patentapplications are hereby incorporated by reference in their entireties.

TECHNICAL FIELD

This patent document relates to video processing techniques, devices andsystems.

BACKGROUND

In spite of the advances in video compression, digital video stillaccounts for the largest bandwidth use on the internet and other digitalcommunication networks. As the number of connected user devices capableof receiving and displaying video increases, it is expected that thebandwidth demand for digital video usage will continue to grow.

SUMMARY

Devices, systems and methods related to digital video processing, andfor example, to non-linear adaptive loop filtering for video processingare described. The described methods may be applied to both the existingvideo coding standards (e.g., High Efficiency Video Coding (HEVC)) andfuture video coding standards (e.g., Versatile Video Coding (VVC)) orcodecs.

In one representative aspect, the disclosed technology may be used toprovide a method for video processing. This method includes encoding avideo unit of a video as an encoded video unit; generatingreconstruction samples from the encoded video unit; performing aclipping operation on the reconstruction samples, wherein a clippingparameter used in the clipping operation is a function of a clippingindex and a bit-depth of the reconstruction samples or a bit-depth ofsamples of the video unit; applying a non-linear adaptive loop filter toan output of the clipping operation; and generating a codedrepresentation of the video using the encoded video unit.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes parsing a codedrepresentation of a video for an encoded video unit representing a videounit of the video; generating reconstruction samples of the video unitfrom the encoded video unit; performing a clipping operation on thereconstruction samples, wherein a clipping parameter used in theclipping operation is a function of a clipping index and a bit-depth ofthe reconstruction samples or a bit-depth of the video unit; andapplying a non-linear adaptive loop filter to an output of the clippingoperation to generate a final decoded video unit.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video; and determining a clipping parameterfor filtering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the determining is based oncoded information of the video and/or the video region and/or the videounit.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video; and determining a clipping parameterfor filtering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the clipping parameter is afunction of a color representation format.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video; and determining a clipping parameterfor filtering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the clipping parameterdepends on whether an in-loop reshaping (ILR) is applied forreconstructing the video unit based on a representation of the videounit in a first domain and a second domain and/or scaling chroma residueof a chroma video unit.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video, wherein the coded representationincludes first side information that provides a clipping parameter forfiltering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter; wherein the first side information issignaled together with second side information indicative of filtercoefficients used in the non-linear adaptive loop filter.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video, wherein the coded representationincludes side information indicative of multiple sets of clippingparameters for filtering a reconstruction of a video unit of a videoregion using a non-linear adaptive loop filter.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video; wherein the coded representationincludes side information that provides one or more clipping parametersfor filtering a reconstruction of a chroma video unit of a video regionusing a non-linear adaptive loop filter, wherein the one or moreclipping parameters depend on a color format.

In another aspect, the disclosed technology may be used to provide amethod for video processing. This method includes performing aconversion between a coded representation of a video comprising one ormore video regions and the video, wherein the coded representationincludes side information that provides a clipping parameter forfiltering a reconstruction of a video unit of a video region using anadaptive loop filter, wherein the performing includes generating afiltered video unit by applying a clipping operation to sampledifferences at a video region level.

In yet another representative aspect, the above-described method isembodied in the form of processor-executable code and stored in acomputer-readable program medium.

In yet another representative aspect, a device that is configured oroperable to perform the above-described method is disclosed. The devicemay include a processor that is programmed to implement this method.

In yet another representative aspect, one or more of the above-disclosedmethods can be an encoder-side implementation or a decoder-sideimplementation.

The above and other aspects and features of the disclosed technology aredescribed in greater detail in the drawings, the description and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of an encoder block diagram for video coding.

FIGS. 2A, 2B and 2C show examples of geometry transformation-basedadaptive loop filter (GALF) filter shapes.

FIG. 3 shows an example of a flow graph for a GALF encoder decision.

FIGS. 4A to 4D show example subsampled Laplacian calculations foradaptive loop filter (ALF) classification.

FIG. 5 shows an example of a luma filter shape.

FIG. 6 shows an example of region division of a Wide Video Graphic Array(WVGA) sequence.

FIG. 7 shows an exemplary flowchart of decoding flow with reshaping.

FIGS. 8A to 8C show flowcharts of example methods for video processingin accordance with some implementations of the disclosed technology.

FIG. 9 shows a flowchart of an example method for video processing inaccordance with some implementations of the disclosed technology.

FIGS. 10A and 10B are block diagrams of examples of a hardware platformfor implementing a video processing technique described in the presentdocument.

DETAILED DESCRIPTION

Due to the increasing demand of higher resolution video, video codingmethods and techniques are ubiquitous in modern technology. Video codecstypically include an electronic circuit or software that compresses ordecompresses digital video, and are continually being improved toprovide higher coding efficiency. A video codec converts uncompressedvideo to a compressed format or vice versa. There are complexrelationships between the video quality, the amount of data used torepresent the video (determined by the bit rate), the complexity of theencoding and decoding algorithms, sensitivity to data losses and errors,ease of editing, random access, and end-to-end delay (latency). Thecompressed format usually conforms to a standard video compressionspecification, e.g., the High Efficiency Video Coding (HEVC) standard(also known as H.265 or MPEG-H Part 2), the Versatile Video Coding (VVC)standard to be finalized, or other current and/or future video codingstandards.

In some embodiments, future video coding technologies are explored usinga reference software known as the Joint Exploration Model (JEM). In JEM,sub-block based prediction is adopted in several coding tools, such asaffine prediction, alternative temporal motion vector prediction(ATMVP), spatial-temporal motion vector prediction (STMVP),bi-directional optical flow (BIO), Frame-Rate Up Conversion (FRUC),Locally Adaptive Motion Vector Resolution (LAMVR), Overlapped BlockMotion Compensation (OBMC), Local Illumination Compensation (LIC), andDecoder-side Motion Vector Refinement (DMVR).

Embodiments of the disclosed technology may be applied to existing videocoding standards (e.g., HEVC, H.265) and future standards to improveruntime performance. Section headings are used in the present documentto improve readability of the description and do not in any way limitthe discussion or the embodiments (and/or implementations) to therespective sections only.

1 Examples of Color Space and Chroma Subsampling

Color space, also known as the color model (or color system), is anabstract mathematical model which simply describes the range of colorsas tuples of numbers, typically as 3 or 4 values or color components(e.g. RGB). Basically speaking, color space is an elaboration of thecoordinate system and sub-space.

For video compression, the most frequently used color spaces are YCbCrand RGB.

YCbCr, Y′CbCr, or Y Pb/Cb Pr/Cr, also written as YCBCR or Y′CBCR, is afamily of color spaces used as a part of the color image pipeline invideo and digital photography systems. Y′ is the luma component and CBand CR are the blue-difference and red-difference chroma components. Y′(with prime) is distinguished from Y, which is luminance, meaning thatlight intensity is nonlinearly encoded based on gamma corrected RGBprimaries.

Chroma subsampling is the practice of encoding images by implementingless resolution for chroma information than for luma information, takingadvantage of the human visual system's lower acuity for colordifferences than for luminance.

1.1 The 4:4:4 Color Format

Each of the three Y′CbCr components have the same sample rate, thusthere is no chroma subsampling. This scheme is sometimes used inhigh-end film scanners and cinematic post production.

1.2 The 4:2:2 Color Format

The two chroma components are sampled at half the sample rate of luma,e.g. the horizontal chroma resolution is halved. This reduces thebandwidth of an uncompressed video signal by one-third with little to novisual difference.

1.3 The 4:2:0 Color Format

In 4:2:0, the horizontal sampling is doubled compared to 4:1:1, but asthe Cb and Cr channels are only sampled on each alternate line in thisscheme, the vertical resolution is halved. The data rate is thus thesame. Cb and Cr are each subsampled at a factor of 2 both horizontallyand vertically. There are three variants of 4:2:0 schemes, havingdifferent horizontal and vertical siting.

-   -   In MPEG-2, Cb and Cr are cosited horizontally. Cb and Cr are        sited between pixels in the vertical direction (sited        interstitially).    -   In JPEG/JFIF, H.261, and MPEG-1, Cb and Cr are sited        interstitially, halfway between alternate luma samples.    -   In 4:2:0 DV, Cb and Cr are co-sited in the horizontal direction.        In the vertical direction, they are co-sited on alternating        lines.

2 Examples of the Coding Flow of a Typical Video Codec

FIG. 1 shows an example of encoder block diagram of VVC. As shown inFIG. 1, the flow of video encoding may start with an input video thatundergoes intra prediction and/or motion estimation/motion compensation(ME/MC). These operations use a feedback from a reconstructed copy ofpreviously coded portion of video. The outputs of intra predictionand/or ME/MC are differentially processed through a transform operation(T), followed by a quantization operation (Q), which is entropy codedinto an output coded representation. In the feedback loop, the encodedrepresentation (output of Q block) may go through an inversequantization operation (IQ), followed by an inverse transform (IT)operation to generate reconstruction samples of the encoded videoblocks.

The reconstructed samples may further be processed through various“in-loop” filtering operations to generate reference samples, orreference blocks, or reference pictures used for further encoding. Thein-loop filtering process chain contains three in-loop filtering blocks:deblocking filter (DF), sample adaptive offset (SAO) and ALF. Unlike DF,which uses predefined filters, SAO and ALF utilize the original samplesof the current picture to reduce the mean square errors between theoriginal samples and the reconstructed samples by adding an offset andby applying a finite impulse response (FIR) filter, respectively, withcoded side information signaling the offsets and filter coefficients.ALF is located at the last processing stage of each picture and can beregarded as a tool trying to catch and fix artifacts created by theprevious stages.

At the decoder side, several of the encoding operations are performed ina reverse order to generate decoded and reconstructed video samples. Forexample, referring to FIG. 1, a decoder may parse the codedrepresentation that is the output of the entropy coding and obtainencode units or blocks of video that are then passed through the inversequantization (IQ) operation and inverse transformation (IT) operation togenerate reconstruction samples of the video unit. The final decodedvideo unit may be generated by applying the in-loop filtering operationsas described above with respect to the feedback loop of the videoencoder. Examples of a geometry transformation-based adaptive loopfilter in JEM

In the JEM, a geometry transformation-based adaptive loop filter (GALF)with block-based filter adaption is applied. For the luma component, oneamong 25 filters is selected for each 2×2 block, based on the directionand activity of local gradients.

3.1 Examples of Filter Shape

In the JEM, up to three diamond filter shapes (as shown in FIGS. 2A, 2Band 2C for the 5×5 diamond, 7×7 diamond and 9×9 diamond, respectively)can be selected for the luma component. An index is signalled at thepicture level to indicate the filter shape used for the luma component.For chroma components in a picture, the 5×5 diamond shape is alwaysused.

3.1.1 Block Classification

Each 2×2 block is categorized into one out of 25 classes. Theclassification index C is derived based on its directionality D and aquantized value of activity Â, as follows:

C=5D+Â.   (1)

To calculate D and A, gradients of the horizontal, vertical and twodiagonal direction are first calculated using 1-D Laplacian:

$\begin{matrix}{{g_{v} = {\sum\limits_{k = {i - 2}}^{i + 3}{\sum\limits_{l = {j - 2}}^{j + 3}V_{k,l}}}},{V_{k,l} = {{{2{R\left( {k,l} \right)}} - {R\left( {k,{l - 1}} \right)} - {R\left( {k,{l + 1}} \right)}}}},} & (2) \\{{g_{h} = {\sum\limits_{k = {i - 2}}^{i + 3}{\sum\limits_{l = {j - 2}}^{j + 3}H_{k,l}}}},{H_{k,l} = {{{2{R\left( {k,l} \right)}} - {R\left( {{k - 1},l} \right)} - {R\left( {{k + 1},l} \right)}}}},} & (3) \\{{g_{d1} = {\sum\limits_{k = {i - 2}}^{i + 3}{\sum\limits_{l = {j - 3}}^{j + 3}{D1_{k,l}}}}},{{D\; 1_{k,l}} = {{{2{R\left( {k,l} \right)}} - {R\left( {{k - 1},{l - 1}} \right)} - {R\left( {{k + 1},{l + 1}} \right)}}}}} & (4) \\{{g_{d2} = {\sum\limits_{k = {i - 2}}^{i + 3}{\sum\limits_{j = {j - 2}}^{j + 3}{D2_{k,l}}}}},{{D\; 2_{k,l}} = {{{2{R\left( {k,l} \right)}} - {R\left( {{k - 1},{l + 1}} \right)} - {R\left( {{k + 1},{l - 1}} \right)}}}}} & (5)\end{matrix}$

Indices i and j refer to the coordinates of the upper left sample in the2×2 block and R(i,j) indicates a reconstructed sample at coordinate(i,j).Then D maximum and minimum values of the gradients of horizontal andvertical directions are set as:

g_(h,v) ^(max)=max(g_(h), g_(v)), g_(h,v) ^(min)=min(g_(h), g_(v)),  (6)

and the maximum and minimum values of the gradient of two diagonaldirections are set as:

g_(d0,d1) ^(max)=max(g_(do), g_(d1)), g_(d0,d1) ^(min)>min(g_(do),g_(d1)),   (7)

To derive the value of the directionality D, these values are comparedagainst each other and with two thresholds t₁ and t₂:Step 1. If both g_(h,v) ^(max)≤t₁·g_(h,v) ^(min) and g_(d0,d1)^(max)≤t₁·g_(d0,d1) ^(min) are true, D is set to 0.Step 2. If g_(h,v) ^(max)/g_(h,v) ^(min)>g_(d0,d1) ^(max)/g_(d0,d1)^(min), continue from Step 3; otherwise continue from Step 4.Step 3. If g_(h,v) ^(max)>t₂·g_(h,v) ^(min), D is set to 2; otherwise Dis set to 1.Step 4. If g_(d0,d1) ^(max)>t₂·g_(d0,d1) ^(min), D is set to 4;otherwise D is set to 3.The activity value A is calculated as:

$\begin{matrix}{{A = {\sum\limits_{k = {i - 2}}^{i + 3}{\sum\limits_{l = {j - 2}}^{j + 3}\left( {V_{k,l} + H_{k,l}} \right)}}}.} & (8)\end{matrix}$

A is further quantized to the range of 0 to 4, inclusively, and thequantized value is denoted as A. For both chroma components in apicture, no classification method is applied, i.e. a single set of ALFcoefficients is applied for each chroma component.

3.1.2 Geometric Transformations of Filter Coefficients

Before filtering each 2×2 block, geometric transformations such asrotation or diagonal and vertical flipping are applied to the filtercoefficients f(k,l) depending on gradient values calculated for thatblock. This is equivalent to applying these transformations to thesamples in the filter support region. The idea is to make differentblocks to which ALF is applied more similar by aligning theirdirectionality.

Three geometric transformations, including diagonal, vertical flip androtation are introduced:

Diagonal: f_(D)(k,l)=f(l, k),

Vertical flip: f_(V)(k, l)=f(k, K−l−1),

Rotation: f_(R)(k, l)=f(k, K−l−1, k),   (9)

Herein, K is the size of the filter and 0≤k, l≤K−1 are coefficientscoordinates, such that location (0,0) is at the upper left corner andlocation (K−1,K−1) is at the lower right corner. The transformations areapplied to the filter coefficients f(k,l) depending on gradient valuescalculated for that block. The relationship between the transformationand the four gradients of the four directions are summarized in Table 1.

TABLE 1 Mapping of the gradient calculated for one block and thetransformations Gradient values Transformation g_(d2) < g_(d1) and g_(h)< g_(v) No transformation g_(d2) < g_(d1) and g_(v) < g_(h) Diagonalg_(d1) < g_(d2) and g_(h) < g_(v) Vertical flip g_(d1) < g_(d2) andg_(v) < g_(h) Rotation

3.1.3 Signaling of Filter Parameters

In the JEM, GALF filter parameters are signaled for the first CTU, i.e.,after the slice header and before the SAO parameters of the first CTU.Up to 25 sets of luma filter coefficients could be signaled. To reducebits overhead, filter coefficients of different classification can bemerged. Also, the GALF coefficients of reference pictures are stored andallowed to be reused as GALF coefficients of a current picture. Thecurrent picture may choose to use GALF coefficients stored for thereference pictures, and bypass the GALF coefficients signaling. In thiscase, only an index to one of the reference pictures is signaled, andthe stored GALF coefficients of the indicated reference picture areinherited for the current picture.

To support GALF temporal prediction, a candidate list of GALF filtersets is maintained. At the beginning of decoding a new sequence, thecandidate list is empty. After decoding one picture, the correspondingset of filters may be added to the candidate list. Once the size of thecandidate list reaches the maximum allowed value (i.e., 6 in currentJEM), a new set of filters overwrites the oldest set in decoding order,and that is, first-in-first-out (FIFO) rule is applied to update thecandidate list. To avoid duplications, a set could only be added to thelist when the corresponding picture doesn't use GALF temporalprediction. To support temporal scalability, there are multiplecandidate lists of filter sets, and each candidate list is associatedwith a temporal layer. More specifically, each array assigned bytemporal layer index (TempIdx) may compose filter sets of previouslydecoded pictures with equal to lower TempIdx. For example, the k-tharray is assigned to be associated with TempIdx equal to k, and it onlycontains filter sets from pictures with TempIdx smaller than or equal tok. After coding a certain picture, the filter sets associated with thepicture will be used to update those arrays associated with equal orhigher TempIdx.

Temporal prediction of GALF coefficients is used for inter coded framesto minimize signaling overhead. For intra frames, temporal prediction isnot available, and a set of 16 fixed filters is assigned to each class.To indicate the usage of the fixed filter, a flag for each class issignaled and if required, the index of the chosen fixed filter. Evenwhen the fixed filter is selected for a given class, the coefficients ofthe adaptive filter f(k,l) can still be sent for this class in whichcase the coefficients of the filter which will be applied to thereconstructed image are sum of both sets of coefficients.

The filtering process of luma component can controlled at CU level. Aflag is signaled to indicate whether GALF is applied to the lumacomponent of a CU. For chroma component, whether GALF is applied or notis indicated at picture level only.

3.1.4 Filtering Process

At decoder side, when GALF is enabled for a block, each sample R(i,j)within the block is filtered, resulting in sample value R′(i,j) as shownbelow, where L denotes filter length, f_(m,n) represents filtercoefficient, and f(k,l) denotes the decoded filter coefficients.

R′(i, j)=Σ_(k=-L/2) ^(L/2) Σ_(l=-L/2) ^(L/2) f(k, l)×R(i+k, j+l)   (10)

3.1.5 Determination Process for Encoder Side Filter Parameters

Overall encoder decision process for GALF is illustrated in FIG. 3. Forluma samples of each CU, the encoder makes a decision on whether or notthe GALF is applied and the appropriate signalling flag is included inthe slice header. For chroma samples, the decision to apply the filteris done based on the picture-level rather than CU-level. Furthermore,chroma GALF for a picture is checked only when luma GALF is enabled forthe picture.

4 Examples of a Geometry Transformation-Based Adaptive Loop Filter inVVC

The current design of GALF in VVC has the following major changescompared to that in JEM:

-   -   1) The adaptive filter shape is removed. Only 7×7 filter shape        is allowed for luma component and 5×5 filter shape is allowed        for chroma component.    -   2) Temporal prediction of ALF parameters and prediction from        fixed filters are both removed.    -   3) For each CTU, one bit flag is signaled whether ALF is enabled        or disabled.    -   4) Calculation of class index is performed in 4×4 level instead        of 2×2. In addition, as proposed in JVET-L0147, sub-sampled        Laplacian calculation method for ALF classification is utilized.        More specifically, there is no need to calculate the        horizontal/vertical/45 diagonal/135 degree gradients for each        sample within one block. Instead, 1:2 subsampling is utilized.

5 Examples of a Region-Based Adaptive Loop Filter in AVS2

ALF is the last stage of in-loop filtering. There are two stages in thisprocess. The first stage is filter coefficient derivation. To train thefilter coefficients, the encoder classifies reconstructed pixels of theluminance component into 16 regions, and one set of filter coefficientsis trained for each category using wiener-hopf equations to minimize themean squared error between the original frame and the reconstructedframe. To reduce the redundancy between these 16 sets of filtercoefficients, the encoder will adaptively merge them based on therate-distortion performance. At its maximum, 16 different filter setscan be assigned for the luminance component and only one for thechrominance components. The second stage is a filter decision, whichincludes both the frame level and LCU level. Firstly the encoder decideswhether frame-level adaptive loop filtering is performed. If frame levelALF is on, then the encoder further decides whether the LCU level ALF isperformed.

5.1 Filter Shape

The filter shape adopted in AVS-2 is a 7×7 cross shape superposing a 3×3square shape, just as illustrated in FIG. 5 for both luminance andchroma components. Each square in FIG. 5 corresponds to a sample.Therefore, a total of 17 samples are used to derive a filtered value forthe sample of position C8. Considering overhead of transmitting thecoefficients, a point-symmetrical filter is utilized with only ninecoefficients left, {C0, C1, . . . , C8}, which reduces the number offilter coefficients to half as well as the number of multiplications infiltering. The point-symmetrical filter can also reduce half of thecomputation for one filtered sample, e.g., only 9 multiplications and 14add operations for one filtered sample.

5.2 Region-Based Adaptive Merge

In order to adapt different coding errors, AVS-2 adopts region-basedmultiple adaptive loop filters for luminance component. The luminancecomponent is divided into 16 roughly-equal-size basic regions where eachbasic region is aligned with largest coding unit (LCU) boundaries asshown in FIG. 6, and one Wiener filter is derived for each region. Themore filters are used, the more distortions are reduced, but the bitsused to encode these coefficients increase along with the number offilters. In order to achieve the best rate-distortion performance, theseregions can be merged into fewer larger regions, which share the samefilter coefficients. In order to simplify the merging process, eachregion is assigned with an index according to a modified Hilbert orderbased on the image prior correlations. Two regions with successiveindices can be merged based on rate-distortion cost.

The mapping information between regions should be signaled to thedecoder. In AVS-2, the number of basic regions is used to represent themerge results and the filter coefficients are compressed sequentiallyaccording to its region order. For example, when {0, 1}, {2, 3, 4}, {5,6, 7, 8, 9} and the left basic regions merged into one regionrespectively, only three integers are coded to represent this merge map,i.e., 2, 3, 5.

5.3 Signaling of Side Information

Multiple switch flags are also used. The sequence switch flag,adaptive_loop_filter_enable, is used to control whether adaptive loopfilter is applied for the whole sequence. The image switch flags,picture_alf_enble[i], control whether ALF is applied for thecorresponding ith image component. Only if the picture_alf_enble[i] isenabled, the corresponding LCU-level flags and filter coefficients forthat color component will be transmitted. The LCU level flags,lcu_alf_enable[k], control whether ALF is enabled for the correspondingkth LCU, and are interleaved into the slice data. The decision ofdifferent level regulated flags is all based on the rate-distortioncost. The high flexibility further makes the ALF improve the codingefficiency much more significantly.

In some embodiments, and for a luma component, there could be up to 16sets of filter coefficients.

In some embodiments, and for each chroma component (Cb and Cr), one setof filter coefficients may be transmitted.

6 GALF in VTM-4

In VTM4.0, the filtering process of the Adaptive Loop Filter, isperformed as follows:

O(x,y)=Σ_((i,j)) w _((i,j)) ·I(x+i,y+j)  (11)

where samples I(x+i,y+j) are input samples, O(x,y) is the filteredoutput sample (i.e. filter result), and w(i,j) denotes the filtercoefficients. In practice, in VTM4.0 it is implemented using integerarithmetic for fixed point precision computations:

$\begin{matrix}{{O\left( {x,y} \right)} = {\left( {{\underset{i = {- \frac{L}{2}}}{\sum\limits^{\frac{L}{2}}}{\underset{j = {- \frac{L}{2}}}{\sum\limits^{\frac{L}{2}}}{{w\left( {i,j} \right)} \cdot {I\left( {{x + i},{y + j}} \right)}}}} + 64} \right) ⪢ 7}} & (12)\end{matrix}$

where L denotes the filter length, and where w(i,j) are the filtercoefficients in fixed point precision.

7 Non-Linear Adaptive Loop Filtering (ALF) 7.1 Filtering Reformulation

Equation (11) can be reformulated, without coding efficiency impact, inthe following expression:

O(x,y)=I(x,y)+Σ_((i,j)≠(0,0)) w(i,j)·(I(x+i,y+j)−I(x,y))  (13)

Herein, w(i,j) are the same filter coefficients as in equation (11)[excepted w(0,0) which is equal to 1 in equation (13) while it is equalto 1−Σ_((i,j)≠(0,0))w(i,j) in equation (11)].

7.2 Modified Filter

Using this above filter formula of (13), we can easily introduce nonlinearity to make ALF more efficient by using a simple clipping functionto reduce the impact of neighbor sample values (I(x+i,y+j)) when theyare too different with the current sample value (I(x,y)) being filtered.

In this proposal, the ALF filter is modified as follows:

O′(x,y)=I(x,y)+Σ_((i,j)≠(0,0)) w(i,j)·K(I(x+i,y+j)−I(x,y),k(i,j))  (14)

Herein, K(d,b)=min(b,max(−b,d)) is the clipping function, and k(i,j) areclipping parameters, which depends on the (i,j) filter coefficient. Theencoder performs the optimization to find the best k(i,j).

In the JVET-N0242 implementation, the clipping parameters k(i,j) arespecified for each ALF filter, one clipping value is signaled per filtercoefficient. It means that up to 12 clipping values can be signalled inthe bitstream per Luma filter and up to 6 clipping values for the Chromafilter.

In order to limit the signaling cost and the encoder complexity, welimit the evaluation of the clipping values to a small set of possiblevalues. In the proposal, we only use 4 fixed values which are the samefor INTER and INTRA tile groups.

Because the variance of the local differences is often higher for Lumathan for Chroma, we use two different sets for the Luma and Chromafilters. We also include the maximum sample value (here 1024 for 10 bitsbit-depth) in each set, so that clipping can be disabled if it is notnecessary.

The sets of clipping values used in the JVET-N0242 tests are provided inthe Table 2. The 4 values have been selected by roughly equallysplitting, in the logarithmic domain, the full range of the samplevalues (coded on 10 bits) for Luma, and the range from 4 to 1024 forChroma.

More precisely, the Luma table of clipping values have been obtained bythe following formula:

$\left. {{AlfClip}_{L} = \left\{ {{{round}\mspace{14mu}\left( \left( (M)^{\frac{1}{N}} \right)^{N - n + 1} \right)\mspace{14mu}{for}\mspace{14mu} n} \in {1\mspace{14mu}\ldots\mspace{14mu} N}} \right\rbrack} \right\},{with}$M = 2¹⁰  and  N = 4.

Similarly, the Chroma tables of clipping values is obtained according tothe following formula:

$\left. {{AlfClip}_{C} = \left\{ {{{round}\mspace{14mu}\left( {A \cdot \left( \left( \frac{M}{A} \right)^{\frac{1}{N - 1}} \right)^{N - n}} \right)\mspace{14mu}{for}\mspace{14mu} n} \in {1\mspace{14mu}\ldots\mspace{14mu} N}} \right\rbrack} \right\},{with}$M = 2¹⁰, N = 4  and  A = 4.

TABLE 2 Authorized clipping values INTRA/INTER tile group LUMA {1024,181, 32, 6} CHROMA {1024, 161, 25, 4}

The selected clipping values are coded in the “alf_data” syntax elementby using a Golomb encoding scheme corresponding to the index of theclipping value in the above Table 2. This encoding scheme is the same asthe encoding scheme for the filter index.

8 In-Loop Reshaping (ILR) in JVET-M0427

The in-loop reshaping (ILR) is also known as Luma Mapping with ChromaScaling (LMCS).

The basic idea of in-loop reshaping (ILR) is to convert the original (inthe first domain) signal (prediction/reconstruction signal) to a seconddomain (reshaped domain).

The in-loop luma reshaper is implemented as a pair of look-up tables(LUTs), but only one of the two LUTs need to be signaled as the otherone can be computed from the signaled LUT. Each LUT is aone-dimensional, 10-bit, 1024-entry mapping table (1D-LUT). One LUT is aforward LUT, FwdLUT, that maps input luma code values Y_(i) to alteredvalues Y_(r):Y_(r)=FwdLUT[Y_(i)]. The other LUT is an inverse LUT,InvLUT, that maps altered code values Y_(r) toŶ_(i):Ŷ_(i)=InvLUT[Y_(r)]. (Ŷ_(i) represents the reconstruction valuesof Y_(i)).

8.1 PWL Model

Conceptually, piece-wise linear (PWL) is implemented in the followingway:

Let x1, x2 be two input pivot points, and y1, y2 be their correspondingoutput pivot points for one piece. The output value y for any inputvalue x between x1 and x2 can be interpolated by the following equation:

y=((y2−y1)/(x2−x1))*(x−x1)+y1

In fixed point implementation, the equation can be rewritten as:

y=((m*x+2FP_PREC−1)>>FP_PREC)+c

Herein, m is scalar, c is an offset, and FP_PREC is a constant value tospecify the precision.

Note that in CE-12 software, the PWL model is used to precompute the1024-entry FwdLUT and InvLUT mapping tables; but the PWL model alsoallows implementations to calculate identical mapping values on-the-flywithout pre-computing the LUTs.

8.2 Test CE12-2 in the 4th VVC Meeting 8.2.1 Luma Reshaping

Test 2 of the in-loop luma reshaping (i.e., CE12-2 in the proposal)provides a lower complexity pipeline that also eliminates decodinglatency for block-wise intra prediction in inter slice reconstruction.Intra prediction is performed in reshaped domain for both inter andintra slices.

Intra prediction is always performed in reshaped domain regardless ofslice type.

With such arrangement, intra prediction can start immediately afterprevious TU reconstruction is done. Such arrangement can also provide aunified process for intra mode instead of being slice dependent. FIG. 7shows the block diagram of the CE12-2 decoding process based on mode.

CE12-2 also tests 16-piece piece-wise linear (PWL) models for luma andchroma residue scaling instead of the 32-piece PWL models of CE12-1.

Inter slice reconstruction with in-loop luma reshaper in CE12-2 (lightershaded blocks indicate signal in reshaped domain: luma residue; intraluma predicted; and intra luma reconstructed)

8.2.2 Luma-Dependent Chroma Residue Scaling

Luma-dependent chroma residue scaling is a multiplicative processimplemented with fixed-point integer operation. Chroma residue scalingcompensates for luma signal interaction with the chroma signal. Chromaresidue scaling is applied at the TU level. More specifically, thefollowing applies:

-   -   For intra, the reconstructed luma is averaged.    -   For inter, the prediction luma is averaged.

The average is used to identify an index in a PWL model. The indexidentifies a scaling factor cScaleInv. The chroma residual is multipliedby that number.

It is noted that the chroma scaling factor is calculated fromforward-mapped predicted luma values rather than reconstructed lumavalues.

8.2.3 Signaling of ILR Side Information

The parameters are (currently) sent in the tile group header (similar toALF). These reportedly take 40-100 bits. The following spec is based onversion 9 of JVET-L1001. The added syntax is highlighted below initalicized font.

In 7.3.2.1 Sequence Parameter Set RBSP Syntax

De- scrip- tor seq_parameter_set_rbsp( ) { sps_seq_parameter_set_idue(v) ... sps_triangle_enabled_flag u(1) sps_ladf_enabled_flag u(1) if (sps_ladf_enabled_flag ) { sps_num_ladf_intervals_minus2 u(2)sps_ladf_lowest_interval_qp_offset se(v) for( i = 0; i <sps_num_ladf_intervals_minus2 + 1; i++ ) { sps_ladf_qp_offset[ i ] se(v)sps_ladf_delta_threshold_minus1[ i ] ue(v) } } sps_reshaper_enabled_flagu(1) rbsp_trailing_bits( ) }

In 7.3.3.1 General Tile Group Header Syntax

De- scrip- tor tile_group_header( ) { ... if(num_tiles_in_tile_group_minus1 > 0 ) { offset_len_minus1 ue(v) for( i =0; i < num_tiles_in_tile_group_minus1; i++ ) entry_point_offset_minus1[i ] u(v) } if ( sps_reshaper_enabled_flag ) {tile_group_reshaper_model_present_flag u(1) if (tile_group_reshaper_model_present_flag ) tile_group_reshaper_model ( )tile _group _reshaper_enable_flag u(1) if (tile_group_reshaper_enable_flag && (!( qtbtt_dual_tree_intra_flag &&tile_group_type == I ) ) ) tile_group _reshaper_chr u(1)oma_residual_scale _flag } byte_alignment( ) }

Add a New Syntax Table Tile Group Reshaper Model:

De- scrip- tor tile_group_reshaper_model ( ) {reshaper_model_min_bin_idx ue(v) reshaper_model_delta_max_bin_idx ue(v)reshaper_model_bin_delta_abs_cw _prec_minus1 ue(v) for ( i =reshaper_model_min_bin_idx; i <= reshaper_model_max_bin_idx; i++ ) {reshape_model_bin_delta_abs_CW [ i ] u(v) if (reshaper_model_bin_delta_abs_CW[ i ] ) > 0 )reshaper_model_bin_delta_sign_CW_flag[ i ] u(1) } }

In General Sequence Parameter Set RBSP Semantics, Add the FollowingSemantics:

sps_reshaper_enabled_flag equal to 1 specifies that reshaper is used inthe coded video sequence (CVS). sps_reshaper_enabled_flag equal to 0specifies that reshaper is not used in the CVS.

In Tile Group Header Syntax, Add the Following Semantics

tile_group_reshaper_model_present_flag equal to 1 specifiestile_group_reshaper_model( ) is present in tile group header.tile_group_reshaper_model_present_flag equal to 0 specifiestile_group_reshaper_model( ) is not present in tile group header. Whentile_group_reshaper_model_present_flag is not present, it is inferred tobe equal to 0.tile_group_reshaper_enabled_flag equal to 1 specifies that reshaper isenabled for the current tile group. tile_group_reshaper_enabled_flagequal to 0 specifies that reshaper is not enabled for the current tilegroup. When tile_group_reshaper_enable_flag is not present, it isinferred to be equal to 0.tile_group_reshaper_chroma_residual_scale_flag equal to 1 specifies thatchroma residual scaling is enabled for the current tile group.tile_group_reshaper_chroma_residual_scale_flag equal to 0 specifies thatchroma residual scaling is not enabled for the current tile group. Whentile_group_reshaper_chroma_residual_scale_flag is not present, it isinferred to be equal to 0.Add tile_group_reshaper_model( ) Syntaxreshape_model_min_bin_idx specifies the minimum bin (or piece) index tobe used in the reshaper construction process. The value ofreshape_model_min_bin_idx shall be in the range of 0 to MaxBinIdx,inclusive. The value of MaxBinIdx shall be equal to 15.reshape_model_delta_max_bin_idx specifies the maximum allowed bin (orpiece) index MaxBinIdx minus the maximum bin index to be used in thereshaper construction process. The value of reshape_model_max_bin_idx isset equal to MaxBinIdx-reshape_model_delta_max_bin_idx.reshaper_model_bin_delta_abs_cw_prec_minusl plus 1 specifies the numberof bits used for the representation of the syntaxreshape_model_bin_delta_abs_CW[i].reshape_model_bin_delta_abs_CW[i] specifies the absolute delta codewordvalue for the ith bin.reshaper_model_bin_delta_sign_CW_flag[i] specifies the sign ofreshape_model_bin_delta_abs_CW[i] as follows:

-   -   If reshape_model_bin_delta_sign_CW_flag[i] is equal to 0, the        corresponding variable RspDeltaCW[i] is a positive value.    -   Otherwise (reshape_model_bin_delta_sign_CW_flag[i] is not equal        to 0), the corresponding variable

RspDeltaCW[i] is a negative value.

When reshape_model_bin_delta_sign_CW_flag[i] is not present, it isinferred to be equal to 0.The variable RspDeltaCW [i=(1 2*reshape_model_bin_delta_sign_CW[i])*reshape_model_bin_delta_abs_CW [i];The variable RspCW[i] is derived as following steps:The variable OrgCW is set equal to (1<<BitDepth_(Y))/(MaxBinIdx+1).

-   -   If reshaper_model_min_bin_idx<=i<=reshaper_model_max_bin_idx        RspCW[i]=OrgCW+RspDeltaCW[i].    -   Otherwise, RspCW[i]=0.        The value of RspCW [i] shall be in the range of 32 to 2*OrgCW−1        if the value of BitDepth_(Y) is equal to 10.        The variables InputPivot[i] with i in the range of 0 to        MaxBinIdx+1, inclusive are derived as follows

InputPivot[i]=i*OrgCW

The variable ReshapePivot[i] with i in the range of 0 to MaxBinIdx+1,inclusive, the variable ScaleCoef[i] and

InvScaleCoeff[i] with i in the range of 0 to MaxBinIdx, inclusive, arederived as follows:

shiftY = 14 ReshapePivot[ 0 ] = 0; for( i = 0; i <= MaxBinIdx ; i++) {ReshapePivot[ i + 1 ] = ReshapePivot[ i ] + RspCW[ i ]  ScaleCoef[ i ] =( RspCW[ i ] * (1 << shiftY) + (1 <<  (Log2(OrgCW) − 1))) >>(Log2(OrgCW))  if ( RspCW[ i ] == 0 ) InvScaleCoeff[ i ] = 0 elseInvScaleCoeff[ i ] = OrgCW * (1 << shiftY) / RspCW[ i ] }The variable ChromaScaleCoef[i] with i in the range of 0 to MaxBinIdx,inclusive, are derived as follows:

-   -   ChromaResidualScaleLut[64]={16384, 16384, 16384, 16384, 16384,        16384, 16384, 8192, 8192, 8192, 8192, 5461, 5461, 5461, 5461,        4096, 4096, 4096, 4096, 3277, 3277, 3277, 3277, 2731, 2731,        2731, 2731, 2341, 2341, 2341, 2048, 2048, 2048, 1820, 1820,        1820, 1638, 1638, 1638, 1638, 1489, 1489, 1489, 1489, 1365,        1365, 1365, 1365, 1260, 1260, 1260, 1260, 1170, 1170, 1170,        1170, 1092, 1092, 1092, 1092, 1024, 1024, 1024, 1024};    -   shiftC=11        -   if (RspCW[i]==0)        -   ChromaScaleCoef[i=(1<<shiftC)        -   Otherwise (RspCW[i] !=0),            ChromaScaleCoef[i=ChromaResidualScaleLut[RspCW[i>>1]

8.2.4 Usage of ILR

At the encoder side, each picture (or tile group) is firstly convertedto the reshaped domain. And all the coding process is performed in thereshaped domain. For intra prediction, the neighboring block is in thereshaped domain; for inter prediction, the reference blocks (generatedfrom the original domain from decoded picture buffer) are firstlyconverted to the reshaped domain. Then the residual are generated andcoded to the bitstream.

After the whole picture (or tile group) finishes encoding/decoding,samples in the reshaped domain are converted to the original domain,then deblocking filter and other filters are applied.

Forward reshaping to the prediction signal is disabled for the followingcases:

-   -   Current block is intra-coded    -   Current block is coded as CPR (current picture referencing, aka        intra block copy, IBC)    -   Current block is coded as combined inter-intra mode (CIIP) and        the forward reshaping is disabled for the intra prediction block

9 Drawbacks of Existing Implementations

The non-linear ALF (NLALF) design in JVET-N0242 has the followingproblems:

-   -   (1) It was designed for 4:2:0 color format. For 4:4:4 color        format, luma and chroma components may be of similar importance.        How to better apply NLALF is unknown.    -   (2) The clipping values are designed for the 10-bit case. How to        define NLALF for other bit-depth hasn't been studied yet.    -   (3) The interaction of in-loop reshaping method and NLALF hasn't        been studied.

10 Exemplary Methods for Improvements in Non-Linear Adaptive LoopFiltering

Embodiments of the presently disclosed technology overcome the drawbacksof existing implementations, thereby providing video coding with highercoding efficiencies. Non-linear adaptive loop filtering, based on thedisclosed technology, may enhance both existing and future video codingstandards, is elucidated in the following examples described for variousimplementations. The examples of the disclosed technology provided belowexplain general concepts, and are not meant to be interpreted aslimiting. In an example, unless explicitly indicated to the contrary,the various features described in these examples may be combined.

-   -   1. It is proposed that the parameters (e.g., clipping parameters        defined in Table 2) used in NLALF may depend on the coded        information.        -   a. It is proposed that the parameters (e.g., clipping            parameters defined in Table 2) used in NLALF may depend on            temporal layer index/low delay check flag/reference            pictures.    -   2. Multiple sets of NLALF parameters may be defined or signaled.        -   a. Alternatively, furthermore, when multiple sets of NLALF            parameters are signaled, they may be signaled in a data unit            such as Adaptation Parameter Set (APS)/tile group            header/video data units.        -   b. In one example, the NLALF parameters are signaled in a            predictive way.            -   i. For example, one set of NLALF parameters signaled in                one data unit (such as APS or tile group, or slice) are                predicted by another set of NLALF parameters signaled in                the same data unit.            -   ii. For example, one set of NLALF parameters signaled in                one data unit (such as APS or tile group, or slice) are                predicted by another set of NLALF parameters signaled in                another data unit.    -   3. It is proposed that the parameters (e.g., clipping parameters        defined in Table 2) used in NLALF may depend on bit-depth of        reconstructed samples before applying NLALF.        -   a. Alternatively, it is proposed that the parameters (e.g.,            clipping parameters defined in Table 2) used in NLALF may            depend on input bit-depth of samples before being            encoded/decoded.        -   b. In one example, the parameters for one given bit-depth            may be derived from that assigned for the other bit-depth.            -   i. In one example, shifting operations according to                bit-depth may be applied to derive the parameters for                one given bit-depth.    -   4. It is proposed that the parameters (e.g., clipping parameters        defined in Table 2) used in NLALF may depend on the color        representation format.        -   a. In one example, for the RGB case, the parameter with same            index for the G color component and for the B/R color            components.    -   5. It is proposed that the parameters (e.g., clipping parameters        defined in Table 2) used in NLALF may depend on whether the        in-loop reshaping (ILR) method is applied.        -   a. In one example, the parameters may be different when ILR            is enabled or disabled.    -   6. It is proposed to store the filter parameters (such as filter        coefficients) and NLALF parameters (such as clipping parameters)        together.        -   a. In one example, both of them may be stored in APS.        -   b. In one example, when one video data unit (e.g.,            CTU/region/tile group) uses filter coefficients associated            with one APS, the associated NLALF parameters may be also            utilized.        -   c. Alternatively, for coding/decoding one video data unit            (e.g., CTU/region/tile group), when prediction from filter            coefficients associated with one APS is enabled, the            associated NLALF parameters may be also utilized for            predicting the NLALF parameters for the one video data unit            from the same APS.    -   7. How to handle NLALF for chroma color components may depend on        the color format.        -   a. In one example, for one given color format (such as            4:4:4), two chroma components may use different NLALF            parameters.    -   8. It is proposed that the clipping in ALF may be turned on or        off at sequence level, picture level, slice level, tile group        level, tile level, CTU level, CU level or block level.        -   a. For example, whether to turn on the clipping in ALF may            be signaled to decoder such as in SPS, PPS, slice header,            tile group header, tile, CTU, CU, or block.

The examples described above may be incorporated in the context of themethod described below, e.g., methods 810 to 840, which may beimplemented at a video decoder or a video encoder.

FIG. 8A shows a flowchart of an exemplary method for video processing.The method 810 includes, at step 812, encoding a video unit of a videoas an encoded video unit. The method 810 further includes, at step 813,generating reconstruction samples from the encoded video unit. Themethod 810 further includes, at step 814, performing a clippingoperation on the reconstruction samples, wherein a clipping parameterused in the clipping operation is a function of a clipping index and abit-depth of the reconstruction samples or a bit-depth of samples of thevideo unit. The method 810 further includes, at step 815, applying anon-linear adaptive loop filter to an output of the clipping operation.The method 810 further includes, at step 816, generating a codedrepresentation of the video using the encoded video unit.

FIG. 8B shows a flowchart of an exemplary method for video processing.The method 820 includes, at step 822, parsing a coded representation ofa video for an encoded video unit representing a video unit of thevideo. The method 820 further includes, at step 823, generatingreconstruction samples of the video unit from the encoded video unit.The method 820 includes, at step 824, performing a clipping operation onthe reconstruction samples, wherein a clipping parameter used in theclipping operation is a function of a clipping index and a bit-depth ofthe reconstruction samples or a bit-depth of the video unit. The method820 further includes, at step 825, applying a non-linear adaptive loopfilter to an output of the clipping operation to generate a finaldecoded video unit.

FIG. 8C shows a flowchart of an exemplary method for video processing.The method 830 includes, at step 832, performing a conversion between acoded representation of a video comprising one or more video regions andthe video. The method 830 further includes, at step 834, determining aclipping parameter for filtering a reconstruction of a video unit of avideo region using a non-linear adaptive loop filter. In someimplementations, the determining is based on coded information of thevideo and/or the video region and/or the video unit. In someimplementations, the clipping parameter is a function of a colorrepresentation format. In some implementations, the clipping parameterdepends on whether an in-loop reshaping (ILR) is applied forreconstructing the video unit based on a representation of the videounit in a first domain and a second domain and/or scaling chroma residueof a chroma video unit.

FIG. 9 shows a flowchart of an exemplary method for video processing.The method 840 includes performing a conversion between a codedrepresentation of a video comprising one or more video regions and thevideo. In some implementations, the coded representation includes firstside information that provides a clipping parameter for filtering areconstruction of a video unit of a video region using a non-linearadaptive loop filter and the first side information is signaled togetherwith second side information indicative of filter coefficients used inthe non-linear adaptive loop filter. In some implementations, the codedrepresentation includes side information indicative of multiple sets ofclipping parameters for filtering a reconstruction of a video unit of avideo region using a non-linear adaptive loop filter. In someimplementations, the coded representation includes side information thatprovides one or more clipping parameters for filtering a reconstructionof a chroma video unit of a video region using a non-linear adaptiveloop filter, wherein the one or more clipping parameters depend on acolor format. In some implementations, the coded representation includesside information that provides a clipping parameter for filtering areconstruction of a video unit of a video region using an adaptive loopfilter, wherein the performing includes generating a filtered video unitby applying a clipping operation to sample differences at a video regionlevel.

11 Example Implementations of the Disclosed Technology

FIG. 10A is a block diagram of a video processing apparatus 900. Theapparatus 900 may be used to implement one or more of the methodsdescribed herein. The apparatus 900 may be embodied in a smartphone,tablet, computer, Internet of Things (IoT) receiver, and so on. Theapparatus 900 may include one or more processors 902, one or morememories 904 and video processing hardware 906. The processor(s) 902 maybe configured to implement one or more methods (including, but notlimited to, method 800) described in the present document. The memory(memories) 904 may be used for storing data and code used forimplementing the methods and techniques described herein. The videoprocessing hardware 906 may be used to implement, in hardware circuitry,some techniques described in the present document.

FIG. 10B is another example of a block diagram of a video processingsystem in which disclosed techniques may be implemented. FIG. 10B is ablock diagram showing an example video processing system 4100 in whichvarious techniques disclosed herein may be implemented. Variousimplementations may include some or all of the components of the system4100. The system 4100 may include input 4102 for receiving videocontent. The video content may be received in a raw or uncompressedformat, e.g., 8 or 10 bit multi-component pixel values, or may be in acompressed or encoded format. The input 4102 may represent a networkinterface, a peripheral bus interface, or a storage interface. Examplesof network interface include wired interfaces such as Ethernet, passiveoptical network (PON), etc. and wireless interfaces such as Wi-Fi orcellular interfaces.

The system 4100 may include a coding component 4104 that may implementthe various coding or encoding methods described in the presentdocument. The coding component 4104 may reduce the average bitrate ofvideo from the input 4102 to the output of the coding component 4104 toproduce a coded representation of the video. The coding techniques aretherefore sometimes called video compression or video transcodingtechniques. The output of the coding component 4104 may be eitherstored, or transmitted via a communication connected, as represented bythe component 4106. The stored or communicated bitstream (or coded)representation of the video received at the input 4102 may be used bythe component 4108 for generating pixel values or displayable video thatis sent to a display interface 4110. The process of generatinguser-viewable video from the bitstream representation is sometimescalled video decompression. Furthermore, while certain video processingoperations are referred to as “coding” operations or tools, it will beappreciated that the coding tools or operations are used at an encoderand corresponding decoding tools or operations that reverse the resultsof the coding will be performed by a decoder.

Examples of a peripheral bus interface or a display interface mayinclude universal serial bus (USB) or high definition multimediainterface (HDMI) or Displayport, and so on. Examples of storageinterfaces include SATA (serial advanced technology attachment), PCI,IDE interface, and the like. The techniques described in the presentdocument may be embodied in various electronic devices such as mobilephones, laptops, smartphones or other devices that are capable ofperforming digital data processing and/or video display.

Some embodiments of the disclosed technology include making a decisionor determination to enable a video processing tool or mode. In anexample, when the video processing tool or mode is enabled, the encoderwill use or implement the tool or mode in the processing of a block ofvideo, but may not necessarily modify the resulting bitstream based onthe usage of the tool or mode. That is, a conversion from the block ofvideo to the bitstream representation of the video will use the videoprocessing tool or mode when it is enabled based on the decision ordetermination. In another example, when the video processing tool ormode is enabled, the decoder will process the bitstream with theknowledge that the bitstream has been modified based on the videoprocessing tool or mode. That is, a conversion from the bitstreamrepresentation of the video to the block of video will be performedusing the video processing tool or mode that was enabled based on thedecision or determination.

Some embodiments of the disclosed technology include making a decisionor determination to disable a video processing tool or mode. In anexample, when the video processing tool or mode is disabled, the encoderwill not use the tool or mode in the conversion of the block of video tothe bitstream representation of the video. In another example, when thevideo processing tool or mode is disabled, the decoder will process thebitstream with the knowledge that the bitstream has not been modifiedusing the video processing tool or mode that was disabled based on thedecision or determination.

In the present document, the term “video processing” may refer to videoencoding, video decoding, video compression or video decompression. Forexample, video compression algorithms may be applied during conversionfrom pixel representation of a video to a corresponding bitstreamrepresentation or vice versa. The bitstream representation of a currentvideo block may, for example, correspond to bits that are eitherco-located or spread in different places within the bitstream, as isdefined by the syntax. For example, a macroblock may be encoded in termsof transformed and coded error residual values and also using bits inheaders and other fields in the bitstream.

In some embodiments, the video coding methods may be implemented usingan apparatus that is implemented on a hardware platform as describedwith respect to FIG. 10A or 10B.

Various techniques and embodiments may be described using the followingclause-based format.

The first set of clauses describe certain features and aspects of thedisclosed techniques listed in the previous section.

1. A method for video processing, comprising: determining, based oncoded information of a current video block, a set of parameters for thecurrent video block; and reconstructing, based on performing anon-linear filtering operation using the set of parameters, the currentvideo block from a corresponding bitstream representation.

2. The method of clause 1, wherein the non-linear filtering operationcomprises a non-linear adaptive loop filtering.

3. The method of clause 1 or 2, wherein the set of parameters comprisesat least one clipping value for a luma component or a chroma componentof the current video block.

4. The method of clause 3, wherein the non-linear filtering operation isbased on a color format of the chroma component.

5. The method of any of clauses 1 to 3, wherein the coded informationcomprises a temporal layer index, a low delay check flag or one or morereference pictures.

6. The method of any of clauses 1 to 3, wherein the coded informationcomprises a bit-depth of reconstructed samples prior to the non-linearfiltering operation.

7. The method of any of clauses 1 to 3, wherein the coded informationcomprises a color representation format.

8. The method of any of clauses 1 to 3, wherein the coded informationcomprises an indication of applying an in-loop reshaping (ILR) method.

9. The method of any of clauses 1 to 3, wherein the correspondingbitstream representation comprises multiple sets of parameters thatinclude the set of parameters, and wherein the multiple sets ofparameters in signaled in an Adaptation Parameter Set (APS), a tilegroup header or one or more video data units.

10. The method of any of clauses 1 to 3, wherein the correspondingbitstream representation comprises an Adaptation Parameter Set (APS)that includes the set of parameters and one or more filter coefficientsassociated with the non-linear filtering operation.

11. The method of clause 1 or 2, wherein the set of parameters comprisesone or more clipping values, and wherein the non-linear filteringoperation is performed at a sequence level, a picture level, a slicelevel, a tile group level, a tile level, a coding tree unit (CTU) level,a coding unit (CU) level or a block level.

12. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of clauses 1 to 11.

13. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of clauses 1 to 11.

The second set of clauses describe certain features and aspects of thedisclosed techniques listed in the previous section, including, forexample, Example Implementations 1 and 3-5.

1. A video processing method, comprising: encoding a video unit of avideo as an encoded video unit; generating reconstruction samples fromthe encoded video unit; performing a clipping operation on thereconstruction samples, wherein a clipping parameter used in theclipping operation is a function of a clipping index and a bit-depth ofthe reconstruction samples or a bit-depth of samples of the video unit;applying a non-linear adaptive loop filter to an output of the clippingoperation; and generating a coded representation of the video using theencoded video unit.

2. The method of clause 1, wherein the clipping index is signaled in thecoded representation.

3. A video processing method, comprising: parsing a coded representationof a video for an encoded video unit representing a video unit of thevideo; generating reconstruction samples of the video unit from theencoded video unit; performing a clipping operation on thereconstruction samples, wherein a clipping parameter used in theclipping operation is a function of a clipping index and a bit-depth ofthe reconstruction samples or a bit-depth of the video unit; andapplying a non-linear adaptive loop filter to an output of the clippingoperation to generate a final decoded video unit.

4. The method of clause 3, wherein the clipping index is determined atleast based on a field in the coded representation.

5. The method of clause 3, wherein the clipping index is determinedusing a pre-defined rule.

6. The method of clause 1 or 3, wherein the function of the clippingindex and the bit-depth of the reconstruction samples or the bit-depthof the video unit is such that the function returns different values fora given value of the clipping index based on the bit-depth of thereconstruction samples or the bit-depth of the video unit.

7. The method of clause 1 or 3, wherein a mapping between the clippingindex and the clipping parameter depends on the bit-depth of thereconstruction sample or the bit-depth of the video unit.

8. The method of clause 1 or 3, wherein a first clipping valuecorresponding to a first clipping index for a given bit-depth is derivedbased on a second clipping value corresponding to a second clippingindex for another bit-depth.

9. The method of clause 8, wherein a shifting operation using anotherbit-depth is applied to derive the clipping parameter for the givenbit-depth.

10. The method of any clauses 1 to 9, wherein the coded representationincludes the clipping parameter that control an upper or lower bound oftwo sample differences used in the non-linear adaptive loop filter.

11. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video; and determining a clipping parameter forfiltering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the determining is based oncoded information of the video and/or the video region and/or the videounit.

12. The method of clause 11, wherein the coded information comprises atemporal layer index.

13. The method of clause 11, wherein the coded information comprises alow delay check flag.

14. The method of clause 11, wherein the coded information comprises oneor more reference pictures.

15. The method of any of clauses 11 to 14, wherein the video regioncomprises a video picture.

16. The method of any of clauses 11 to 14, wherein the video unitcomprises a coding unit.

17. The method of any of clauses 11 to 16, wherein the clippingparameter controls an upper or lower bound of two sample differencesused in the non-linear adaptive loop filter.

18. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video; and determining a clipping parameter forfiltering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the clipping parameter is afunction of a color representation format.

19. The method of clause 18, wherein, for a RGB color format, theclipping parameter has a same index for a green color component and fora blue or red color component.

20. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video; and determining a clipping parameter forfiltering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter, and wherein the clipping parameterdepends on whether an in-loop reshaping (ILR) is applied forreconstructing the video unit based on a representation of the videounit in a first domain and a second domain and/or scaling chroma residueof a chroma video unit.

21. The method of any of clauses 1 to 21, wherein the clipping parametercorresponds to a clipping value for a luma component or a chromacomponent of the video unit.

22. The method of any of clauses 1 to 21, wherein the method furtherincludes, during the conversion, generating a filtered video unit byapplying the non-linear adaptive loop filter to the reconstruction ofthe video unit, and using the filtered video unit for determining aprediction of another video unit of the video.

23. The method of any of clauses 1 to 22, wherein the performing of theconversion includes generating the coded representation from the video.

24. The method of any of clauses 1 to 22, wherein the performing of theconversion includes generating the video from the coded representation.

25. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of clauses 1 to 24.

26. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of clauses 1 to 24.

The third set of clauses describe certain features and aspects of thedisclosed techniques listed in the previous section, including, forexample, Example Implementations 2 and 6-8.

1. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video, wherein the coded representation includes firstside information that provides a clipping parameter for filtering areconstruction of a video unit of a video region using a non-linearadaptive loop filter; wherein the first side information is signaledtogether with second side information indicative of filter coefficientsused in the non-linear adaptive loop filter.

2. The method of clause 1, wherein the first side information and thesecond side information are signaled in a same adaptation parameter set.

3. The method of clause 1, wherein in a case that at least some of thefilter coefficients associated with an adaptation parameter set is usedby a video data unit, the clipping parameter associated with theadaptation parameter set is also used by the video data unit.

4. The method of clause 1, wherein in a case that a prediction from atleast some of the filter coefficients associated with an adaptationparameter set is enabled for the conversion of a video data unit, theparameter associated with the adaptation parameter set is used forpredicting another parameter for another video data unit from theadaptation parameter set.

5. The method of clause 3 or 4, wherein the video data unit is a codingtree unit, a video region, or a tile group.

6. The method of any of clauses 1 to 5, wherein the parametercorresponds to a clipping value for a luma component or a chromacomponent of the video unit.

7. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video, wherein the coded representation includes sideinformation indicative of multiple sets of clipping parameters forfiltering a reconstruction of a video unit of a video region using anon-linear adaptive loop filter.

8. The method of clause 7, wherein the side information includes themultiple sets of clipping parameters.

9. The method of clause 7, wherein the multiple sets of clippingparameters are known to an encoder and a decoder and the sideinformation includes an index to one or more of the multiple sets ofclipping parameters.

10. The method of clause 7, wherein the multiple sets of the clippingparameters are included in a video data unit or a header of the videounit.

11. The method of clause 10, wherein the video data unit includes anadaptation parameter set, a tile group or a slice.

12. The method of clause 8, wherein one set of the multiple sets of theclipping parameters signaled in a data unit is predicted by another setof the clipping parameters signaled in the data unit.

13. The method of clause 7, wherein one set of the multiple sets of theclipping parameters signaled in a data unit is predicted by another setof the clipping parameters signaled in another data unit.

14. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video; wherein the coded representation includes sideinformation that provides one or more clipping parameters for filteringa reconstruction of a chroma video unit of a video region using anon-linear adaptive loop filter, wherein the one or more clippingparameters depend on a color format.

15. The method of clause 14, wherein, for a certain color format, twochroma components use different clipping parameters.

16. The method of clause 15, wherein the certain color format is 4:4:4.

17. The method of any of clauses 1 to 16, wherein the method furtherincludes, during the conversion, generating a filtered video unit byapplying the non-linear adaptive loop filter to the reconstruction ofthe video unit, and using the filtered video unit for determining aprediction of another video unit of the video.

18. A video processing method, comprising: performing a conversionbetween a coded representation of a video comprising one or more videoregions and the video, wherein the coded representation includes sideinformation that provides a clipping parameter for filtering areconstruction of a video unit of a video region using an adaptive loopfilter, wherein the performing includes generating a filtered video unitby applying a clipping operation to sample differences at a video regionlevel.

19. The method of clause 18, wherein the video region level is asequence level, a picture level, a slice level, a tile group level, atile level, a coding tree unit level, a coding unit level, or a blocklevel.

20. The method of clause 18, wherein an indication to enable theclipping operation is signaled in a sequence parameter set (SPS), apicture parameter set (PPS), a slice header, a tile group header, atile, a coding tree unit, a coding unit, or a block.

21. The method of any of clauses 1 to 20, wherein the video region is avideo picture.

22. The method of any of clauses 1 to 20, wherein the video unit is acoding unit or a transform unit or a slice or a coding tree or a codingtree row.

23. The method of any of clauses 1 to 22, wherein the performing of theconversion includes generating the coded representation from the currentblock.

24. The method of any of clauses 1 to 22, wherein the performing of theconversion includes generating the current block from the codedrepresentation.

25. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of clauses 1 to 24.

26. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of clauses 1 to 24.

From the foregoing, it will be appreciated that specific embodiments ofthe presently disclosed technology have been described herein forpurposes of illustration, but that various modifications may be madewithout deviating from the scope of the invention. Accordingly, thepresently disclosed technology is not limited except as by the appendedclaims.

Implementations of the subject matter and the functional operationsdescribed in this patent document can be implemented in various systems,digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of the subject matter described in this specificationcan be implemented as one or more computer program products, i.e., oneor more modules of computer program instructions encoded on a tangibleand non-transitory computer readable medium for execution by, or tocontrol the operation of, data processing apparatus. The computerreadable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing unit” or “dataprocessing apparatus” encompasses all apparatus, devices, and machinesfor processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of nonvolatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, beconsidered exemplary only, where exemplary means an example. As usedherein, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. Additionally, the use of “or” is intended to include“and/or”, unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

What is claimed is:
 1. A method of processing video data, comprising:performing a conversion between a bitstream of a video comprising one ormore video regions and the video, wherein a clipping parameter forfiltering a reconstruction of a first video unit of a video region usinga non-linear adaptive loop filter is derived based on first sideinformation included in the bitstream and a first adaptation parameterset identification, and wherein filtering coefficients for filtering thereconstruction of the first video unit using a non-linear adaptive loopfilter is derived based on second side information included in thebitstream and the first adaptation parameter set identification.
 2. Themethod of claim 1, wherein in a case that at least some of the filteringcoefficients associated with the first adaptation parameter setidentification is used by the first video unit, the clipping parameterassociated with the first adaptation parameter set identification isalso used by the first video unit.
 3. The method of claim 1, wherein ina case that a prediction from data information associated with the firstadaptation parameter set identification is enabled for at least some ofthe filtering coefficients of the first video unit, the data informationassociated with the first adaptation parameter set identification isalso used for predicting the clipping parameter.
 4. The method of claim1, wherein the first video unit is a coding tree unit, a coding treeblock, or a slice.
 5. The method of claim 1, wherein for a first samplein the first video unit, a first filtering index is derived based onmultiple sample differences in different directions.
 6. The method ofclaim 5, wherein the clipping parameter is derived further based on thefirst filtering index.
 7. The method of claim 5, wherein the first videounit is split into multiple M*M video sub-region, and the multiplesample differences in different directions are derived for every M*Mvideo sub-region, and wherein M is equal to 2 or
 4. 8. The method ofclaim 7, wherein the multiple sample differences in different directionsare derived based on 1:N subsampling rate, wherein N is great than
 1. 9.The method of claim 6, wherein the clipping parameter is derived furtherbased on a bit-depth value of the first video unit.
 10. The method ofclaim 1, wherein the conversion includes encoding the video into thebitstream.
 11. The method of claim 2, wherein the conversion includesdecoding the video from the bitstream.
 12. An apparatus for processingvideo data comprising a processor and a non-transitory memory withinstructions thereon, wherein the instructions upon execution by theprocessor, cause the processor to: perform a conversion between abitstream of a video comprising one or more video regions and the video,wherein a clipping parameter for filtering a reconstruction of a firstvideo unit of a video region using a non-linear adaptive loop filter isderived based on first side information included in the bitstream and afirst adaptation parameter set identification, and wherein filteringcoefficients for filtering the reconstruction of the first video unitusing a non-linear adaptive loop filter is derived based on second sideinformation included in the bitstream and the first adaptation parameterset identification.
 13. The apparatus of claim 12, wherein in a casethat at least some of the filtering coefficients associated with thefirst adaptation parameter set identification is used by the first videounit, the clipping parameter associated with the first adaptationparameter set identification is also used by the first video unit. 14.The apparatus of claim 12, wherein in a case that a prediction from datainformation associated with the first adaptation parameter setidentification is enabled for at least some of the filteringcoefficients of the first video unit, the data information associatedwith the first adaptation parameter set identification is also used forpredicting the clipping parameter.
 15. The apparatus of claim 12,wherein the first video unit is a coding tree unit, a coding tree block,or a slice.
 16. The apparatus of claim 12, wherein for a first sample inthe first video unit, a first filtering index is derived based onmultiple sample differences in different directions.
 17. The apparatusof claim 16, wherein the clipping parameter is derived further based onthe first filtering index.
 18. The apparatus of claim 16, wherein thefirst video unit is split into multiple M*M video sub-region, and themultiple sample differences in different directions are derived forevery M*M video sub-region, and wherein M is equal to 2 or
 4. 19. Anon-transitory computer-readable storage medium storing instructionsthat cause a processor to: perform a conversion between a bitstream of avideo comprising one or more video regions and the video, wherein aclipping parameter for filtering a reconstruction of a first video unitof a video region using a non-linear adaptive loop filter is derivedbased on first side information included in the bitstream and a firstadaptation parameter set identification, and wherein filteringcoefficients for filtering the reconstruction of the first video unitusing a non-linear adaptive loop filter is derived based on second sideinformation included in the bitstream and the first adaptation parameterset identification.
 20. A non-transitory computer-readable recordingmedium storing a bitstream of a video which is generated by a methodperformed by a video processing apparatus, wherein the method comprises:generating a bitstream of a video comprising one or more video regions,wherein a clipping parameter for filtering a reconstruction of a firstvideo unit of a video region using a non-linear adaptive loop filter isderived based on first side information included in the bitstream and afirst adaptation parameter set identification, and wherein filteringcoefficients for filtering the reconstruction of the first video unitusing a non-linear adaptive loop filter is derived based on second sideinformation included in the bitstream and the first adaptation parameterset identification.